This invention relates to data processing systems, methods and computer program products and more particularly to image processing systems, methods and computer program products.
Hemispherical optical projection systems and methods, i.e. systems and methods which project images at an angle of at least about 160 degrees, are used to project images onto the inner surfaces of domes. Hemispherical optical projection systems and methods have long been used in planetariums, commercial and such as OMNIMAX(copyright) theaters. With the present interest in virtual reality, hemispherical optical projection systems and methods have been investigated for projecting images which simulate a real environment. Such images are typically computer-generated multimedia images Including video, but they may also be generated using film or other media. Home theater has also generated much interest, and hemispherical optical projection systems and methods are also being investigated for home theater applications.
Alternate Realities Corporation, the assignee of the present invention, has embarked in a program of research and development to produce a low-cost, versatile, portable, high quality hemispherical optical projection system. In application Ser. No. 08/593,041, entitled xe2x80x9cMulti-Pieced, Portable Projection Dome and Method of Assembling the Samexe2x80x9d to Zobel, Jr. et al., which is assigned to the assignee of the and building a low-cost, portable dome are described. In application Ser. No. 08/593,699, entitled xe2x80x9cTiltable Hemispherical Optical Projection Systems and Methods having Constant Angular Separation of Projected Pixelsxe2x80x9d to Colucci et al., optical systems and methods which are optimized for hemispherical optical projection are described.
A major problem of hemispherical optical projection systems and methods is the provision of appropriate data for display on a dome without multimedia images including video, and images which use film or other media, are generally generated for computer display, or theater display. It will be understood by those having skill in the art that although some of those displays may deviate slightly from absolute planarity, the image data is assumed to be projected onto a planar display.
In the field of image processing, hardware and software has been developed for planar image graphics computer systems which convert three-dimensional image data to planar image data for display on a planar display. Manufacturers of planar image graphics computer systems include Silicon Graphics Incorporated (the Infinite Reality or Onyx lines of systems), Sun Microsystems (the SPARC Station and Ultra lines of systems), IBM (the RISC System 6000 Series and Power lines of systems), Hewlett-Packard (the Visualize, Freedom and PRO Vision lines of systems) and many others. These systems process image data for display on a two-dimensional display at speeds which are generally higher than those of general purpose computers or workstations. The above-noted manufacturers and others have now standardized their planar image graphics computer systems (hardware and software) to operate under a standard architecture, referred to as OpenGL(copyright). OpenGL is a programming graphics on platforms from many manufacturers. Over, four hundred commands are provided which can be used to display shapes, compose animated three-dimensional scenes, complete with lighting, anti-aliasing and texture-mapping.
Because OpenGL is an industry standard, programs and libraries written on one platform can be ported easily to another. Moreover, because most of the complex math which is necessary for producing interactive three-dimensional applications is hidden within the commands, developers are relieved of having to decipher lengthy formulas in order to render images on a planar display. The OpenGL architecture is described in publications entitled xe2x80x9c3D Graphics programming with OpenGL (copyright)xe2x80x9d, Clayton Wallnum, 1995, Que Corporation; xe2x80x9cOpenGL(copyright) Reference Manual, The Official Reference Document for OpenGL(copyright) Release 1xe2x80x9d, OpenGL(copyright) Architecture Review Board, sponsoring editor David Rogelberg, 1992; and xe2x80x9cOpenGL(copyright) Programming Guide, The Official Guide to Learning OpenGL(copyright), Release 1xe2x80x9d, OpenGL(copyright) Architecture Review Board, Jackie Knighter, Doni Davis and Nason Woo, 1993, the disclosures of all of which are hereby incorporated herein by reference.
Unfortunately, image data produced by image graphics computer systems which are designed for planar displays will generally be distortion corrected upon projection on a nonplanar display. For example, as images are scaled, translated and/or rotated, they generally will become distorted in shape when they are projected on a nonplanar display such as a dome. This distortion can be a major stumbling block in the acceptance of a dome as a replacement for a planar display.
Many approaches can be taken to correct these distortions. One approach would be to design a nonplanar image graphics computer system which can produce nonplanar images at high speed. Unfortunately, such a hardware and software development effort would likely require many millions of dollars to implement, as evidenced by the massive development efforts which OpenGL.
Another approach would be to require all content producers to modify their data to accommodate the unique environment of a dome. Unfortunately, this requirement would also likely hinder the acceptance of a dome among content providers.
Yet another alternative would be to design and/or port a computer program, which can operate on a general purpose personal computer or workstation, to provide nonplanar image data processing. Unfortunately, such a program would also likely require large development and/or porting efforts and would likely be slower than state-of-the-art image processing systems using planar image graphics computer systems such as OpenGL systems.
Still another alternative would be to divide the nonplanar display into a large number of approximately planar segments and use a plurality of planar image display systems to produce the nonplanar display. Such a solution is often referred to as xe2x80x9ctilingxe2x80x9d, and is often used in flight simulators and wide-angle theaters. Unfortunately, multiple projection systems may be costly and difficult to set up and align and may still include distortions because the display surfaces are not truly planar.
The above description has focused on hemispherical displays. However, there is also interest in many other forms of nonplanar displays wherein wide angle projection of an image is generated using a single image pipeline. For example, virtual reality helmets may use a nonplanar, nonhemispherical viewing area. In other applications, images may be projected onto cylindrical projection surfaces. In yet other applications, images may be projected on the inside of a cube such as a room. All of these display applications may encounter the problems described above in attempting to reduce distortion upon projection of image data on a nonplanar display.
In view of the above problems, it is therefore an object of the present invention to provide improved methods, systems (apparatus) and computer program products for converting image data to nonplanar image data for display on a nonplanar display.
It is another object of the present invention to provide methods, systems and computer program products for converting image data to nonplanar image data which do not require development of entirely new hardware/software image processing architectures.
These and other objects are provided according to the present invention by methods, systems and computer program products for converting image data to nonplanar image data wherein the image data is nonplanarly distortion corrected to produce nonplanar image data and the nonplanar image data is input to a planar image graphics computer system which converts image data to planar image data for display on a planar display, for processing by the planar image graphics computer system. Accordingly, the image data is distortion corrected and manipulated as necessary, so that the distortion corrected image data can then be applied to a standard planar image graphics computer system such as an OpenGL system. The power of the OpenGL system can then be used for image processing. It is therefore not necessary to develop a new architecture for nonplanar image processing. Moreover, the large hardware and software investment in OpenGL and other planar image processing architectures can be used advantageously, and the nonplanar image processing may be noninvasive to OpenGL capabilities.
In particular, it will be understood by those having skill in the art that a planar image data processing system generally employs a transformation matrix which is generally a 4xc3x974 matrix which is used to transform vectors and other fundamental geometric primitives corresponding to vertices of objects using matrix multiplication. According to the invention, a transformation matrix is generated and obtained from the planar image graphics computer system. A plurality of vertices of the image data is then matrix multiplied by the obtained transformation matrix, to produce transformed image data. The transformed image data is then nonplanarly distortion corrected to produce nonplanar image data. A passthrough transformation computer system. For example, an identity matrix may be provided. The nonplanar image data is then input to processing by the planar image graphics computer system.
Thus, the matrix multiplication is performed outside the planar image graphics computer system using a transformation matrix which is generated by the planar image data computer system. However, remaining graphics processing is preferably performed using the high-speed hardware and software of the planar image graphics computer system. The power of the planar image graphics computer system is thereby used, without requiring content providers to modify their image data.
As described above, the transformation matrix is generally a 4xc3x974 matrix, each element of which is generally a floating point number. Accordingly, the obtained of transformation matrices from the planar image graphics computer system, using subroutine calls or other conventional techniques, may slow down processing speed dramatically. According to an aspect of the invention, it has been found that the transformation matrix need only be obtained once for each object in the image data. Moreover, in other cases, the transformation matrix need only be obtained once for a plurality of objects in the image data. Accordingly, high speed processing may be provided.
According to an embodiment of the present invention, a pair of flags are used to govern the obtaining of the transformation matrix. In particular, a first flag keeps track of which transformation matrix is being operated upon. As is well known to those having skill in the art, there are generally two transformation matrices which are manipulated in planar image graphics computer systems: a projection_view matrix and a modelview matrix. Each of these matrices is generally a 4xc3x974 matrix. The projection_view matrix is used to perform planar graphical operations which project images onto a planar display. The modelview matrix is used when the projection is tile same but the viewpoint of the observer is varying. Other transformation matrices may also be used.
The second flag is used to keep track of when to the invention, a new transformation matrix is preferably not calculated more often than it needs to be calculated. Accordingly, if the transformation matrix is being applied to a single object, the transformation matrix may be used for all vertices of that object. Thus, for example, if a single object is being rotated, the transformation matrix may be used for all vertices of that object, as determined by the second flag. Moreover, if a plurality of objects are being rotated, for example due to rotation of the viewpoint, the transformation matrix may be used for all vertices of all of the objects. The transformation matrix is preferably not recalculated, since replacing the existing transformation matrix in the planar image graphics computer system, recalculating the new matrix and then obtaining the recalculated. Matrix from the planar image graphics system. The operations of fetching, storing and matrix multiplying of double floating point numbers may be time consuming. The present invention can thus provide high speed processing of coplanar images, using standard planar image processing systems.